Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How do I extract the features using an FPN based model? #7

Open
kevkid opened this issue Jun 16, 2020 · 8 comments
Open

How do I extract the features using an FPN based model? #7

kevkid opened this issue Jun 16, 2020 · 8 comments

Comments

@kevkid
Copy link

kevkid commented Jun 16, 2020 •
edited
Loading

❓ How to use Detectron2

Questions like:

  1. How to do extract the features with detectron2 as shown in your demo? I am using this config
    I have a pretrained model (trained on custom data) and would like to be able to extract the features of the bounding boxes.
    I keep getting
    AttributeError: 'StandardROIHeads' object has no attribute '_shared_roi_transform'

I apologized if the answer is obvious, I am very new to object detection.

Thank you!

EDIT

I am confused, I have detectron2 installed. Can I install this onto of my current installation and use it? I ask because it looks like there is a fork of detectron2 in this package which is different from the original detectron2. If I train a model using vanilla detectron2 and install this, can I just load the model weights, and extract the features from there?
@airsplay
Copy link
Owner

I think that you could load the D2 weight into this repo but I am not sure about its validity. Since I have changed the files to support a specific version of faster rcnn in BUTD.

BTW, the modifications are listed in the demo where the model is loaded.

@faizanahemad
Copy link

@kevkid You might be facing this issue since FB/detectron2 has moved a lot and there are too many changes there to maintain compatibility. I ported changes from airsplay/py-bottom-up-attention to most recent detectron2 master over https://github.com/faizanahemad/detectron2 .
You can install that as python -m pip install 'git+https://github.com/faizanahemad/detectron2.git' .
Follow the demo or it's demo.py file version which does the same as airsplay's demo.

@kevkid
Copy link
Author

kevkid commented Jul 3, 2020

@faizanahemad I wrote this up to extract the features for a detectron2 model. I believe this is correct and I have lots of comments. Let me know if this looks right to you:

class feature_extractor:
    '''
    Feature Extractor for detectron2
    '''
    def __init__(self, path = None, output_folder='./output', model = None, pred_thresh = 0.5):
        self.pred_thresh = pred_thresh
        self.output_folder = output_folder
        assert path is not None, 'Path should not be none'
        self.path = path
        if model == None:
            self.model = self._build_detection_model()
        else:
            assert model == detectron2.engine.defaults.DefaultPredictor, "model should be 'detectron2.engine.defaults.DefaultPredictor'"#
            self.model = model
            self.model.eval()
    def _build_detection_model(self):
        cfg = get_cfg()
        cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))
        cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
        cfg.SOLVER.IMS_PER_BATCH = 1
        cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1  # only has one class (pnumonia)
        #Just run these lines if you have the trained model im memory
        cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = self.pred_thresh   # set the testing threshold for this model
        #build model and return
        return DefaultPredictor(cfg)
    def _process_feature_extraction(self, img):#step 3
        '''
        #predictor.model.roi_heads.box_predictor.test_topk_per_image = 1000
        #predictor.model.roi_heads.box_predictor.test_nms_thresh = 0.99
        #predictor.model.roi_heads.box_predictor.test_score_thresh = 0.0
        #pred_boxes = [x.pred_boxes for x in instances]#can use prediction boxes
        '''
        torch.cuda.empty_cache()
        predictor = self.model
        with torch.no_grad():#https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html : _forward_box()
            features = predictor.model.backbone(img.tensor)#have to unsqueeze
            proposals, _ = predictor.model.proposal_generator(img, features, None)
            results, _ = predictor.model.roi_heads(img, features, proposals, None)
            #instances = predictor.model.roi_heads._forward_box(features, proposals)
        #get proposed boxes + rois + features + predictions
        proposal_boxes = [x.proposal_boxes for x in proposals]
        proposal_rois = predictor.model.roi_heads.box_pooler([features[f] for f in predictor.model.roi_heads.in_features], proposal_boxes)
        box_features = predictor.model.roi_heads.box_head(proposal_rois)
        predictions = predictor.model.roi_heads.box_predictor(box_features)#found here: https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html
        #WE CAN USE THE PREDICTION CLS TO FIND TOP SCOREING PROPOSAL BOXES!
        #pred_instances, losses = predictor.model.roi_heads.box_predictor.inference(predictions, proposals)#something to do with: NMS threshold for prediction results. found: https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/roi_heads/fast_rcnn.py#L460
        pred_df = pd.DataFrame(predictions[0].softmax(-1).tolist())
        pred_classes = pred_df.iloc[:,:-1].apply(np.argmax, axis=1)#get predicted classes
        keep = pred_df[pred_df.iloc[:,:-1].apply(lambda x: (x > self.pred_thresh)).values].index.tolist()#list of instances we should keep
        #start subsetting
        box_features = box_features[keep]
        proposal_boxes = proposals[0].proposal_boxes[keep]
        pred_classes = pred_classes[keep]
        probs = pred_df.iloc[keep, :].apply(lambda x: x[np.argmax(x)], axis=1).tolist()

        #['bbox', 'num_boxes', 'objects', 'image_width', 'image_height', 'cls_prob', 'image_id', 'features']
        #img.image_sizes[0]#h, w
        result = {
            'bbox': proposal_boxes.tensor.to('cpu').numpy(),
            'num_boxes' : len(proposal_boxes),
            'objects' : pred_classes.to_numpy,
            #'image_height': img.image_sizes[0][0],
            #'image_width': img.image_sizes[0][1],
            'cls_prob': np.asarray(probs),#needs to turn into vector!!!!!!!!!!
            'features': box_features.to('cpu').detach().numpy()
        }
        return result
            
    def _save_feature(self, file_name, feature, info):
        file_base_name = os.path.basename(file_name)
        file_base_name = file_base_name.split(".")[0]
        feature["image_id"] = file_base_name
        feature['image_height'] = info['height']
        feature['image_width'] = info['width']
        file_base_name = file_base_name + ".npy"
        np.save(os.path.join(self.output_folder, file_base_name), feature)
        
    def extract_features(self):#step 1
        torch.cuda.empty_cache()
        image_dir = self.path
        #print(image_dir)
        if type(image_dir) == pd.core.frame.DataFrame:#or pandas.core.frame.DataFrame. Iterate over a dataframe
            samples = []
            for idx, row in image_dir.iterrows():#get better name
                file = row['path']
                try:
                    features, infos = self.get_detectron2_features([file])
                    self._save_feature(file, features, infos[0])
                    samples.append(row)
                except BaseException:#if no features were found!
                    print('No features were found!')
                    pass
            df = pd.DataFrame(samples)
            #save final csv containing image base names, reports and report locations
            df.to_csv(os.path.join(self.output_folder, 'img_infos.csv'))
        elif os.path.isfile(image_dir):#if its a single file
            features, infos = self.get_detectron2_features([image_dir])
            self._save_feature(image_dir, features[0], infos[0])
            return features, infos
        else:#if its a directory
            files = glob.glob(os.path.join(image_dir, "*"))
            for idx, file in enumerate(files):
                try:
                    features, infos = self.get_detectron2_features([file])
                    self._save_feature(file, features, infos[0])
                except BaseException:
                    print('BaseException')
                    pass

    def get_detectron2_features(self, image_paths):#step 2
        #we have to PREPROCESS the tensor before partially executing it!
        #taken from https://github.com/facebookresearch/detectron2/blob/master/detectron2/engine/defaults.py
        predictor = self.model
        images = []
        image_info = []
        for image_path in image_paths:
            img = cv2.imread(image_path)
            height, width = img.shape[:2]
            img = predictor.transform_gen.get_transform(img).apply_image(img)
            img = torch.as_tensor(img.astype("float32").transpose(2, 0, 1))
            images.append({"image": img, "height": height, "width": width})
            image_info.append({"image_id": os.path.basename(image_path), "height": height, "width": width})
        imageList = predictor.model.preprocess_image(images)
        #returns features and infos
        return self._process_feature_extraction(imageList), image_info

Your input should be a DataFrame with the following columns = ['dicom_id', 'path', 'report_path', 'report', 'study_id', 'y']
Should look something like this:

fe = feature_extractor(path=outputs_df, output_folder='/path/to/your/numpy_files.npy')
fe.extract_features()

@yezhengli-Mr9
Copy link

feature_extractor

@faizanahemad I wrote this up to extract the features for a detectron2 model. I believe this is correct and I have lots of comments. Let me know if this looks right to you:

Hi @kevkid , do not find feature_extractor let alone the "lots of comments" (from github or from current LXMERT repo). Basically, I am interested in using a smaller backbone and hence I am interested in weights conversion (for example, resnet18 instead of resnet101). I also follow #6, #16 .

image

@yezhengli-Mr9
Copy link

yezhengli-Mr9 commented Jan 13, 2021 •
edited
Loading

❓ How to use Detectron2

Questions like:

  1. How to do extract the features with detectron2 as shown in your demo? I am using this config
    I have a pretrained model (trained on custom data) and would like to be able to extract the features of the bounding boxes.
    I keep getting
    AttributeError: 'StandardROIHeads' object has no attribute '_shared_roi_transform'

I apologized if the answer is obvious, I am very new to object detection.

Thank you!

EDIT

I am confused, I have detectron2 installed. Can I install this onto of my current installation and use it? I ask because it looks like there is a fork of detectron2 in this package which is different from the original detectron2. If I train a model using vanilla detectron2 and install this, can I just load the model weights, and extract the features from there?

Hi @kevkid, your mask_rcnn_R_101_FPN_3x.yaml and @ThierryDeruyttere's faster_rcnn_R_101_C4_attr_caffemaxpool.yaml in #16 are starting point for my goal (repeatly, my goal is I am using a smaller backbone (for example, resnet18 instead of resnet101).

@yezhengli-Mr9
Copy link

yezhengli-Mr9 commented Jan 14, 2021 •
edited
Loading

❓ How to use Detectron2

Questions like:

  1. How to do extract the features with detectron2 as shown in your demo? I am using this config
    I have a pretrained model (trained on custom data) and would like to be able to extract the features of the bounding boxes.
    I keep getting
    AttributeError: 'StandardROIHeads' object has no attribute '_shared_roi_transform'

I apologized if the answer is obvious, I am very new to object detection.

Thank you!

EDIT

I am confused, I have detectron2 installed. Can I install this onto of my current installation and use it? I ask because it looks like there is a fork of detectron2 in this package which is different from the original detectron2. If I train a model using vanilla detectron2 and install this, can I just load the model weights, and extract the features from there?

Hi @kevkid , after referring to detectron2_mscoco_proposal_maxnms.py and coming up with code in #6 , I am looking into your feature_extractor and think cfg.MODEL.WEIGHTS = 'R-50.pkl' together with
cfg.merge_from_file('../configs/COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml') or cfg.merge_from_file('../configs/COCO-Detection/faster_rcnn_R_50_DC5_1x.yaml') also result in

AttributeError: 'StandardROIHeads' object has no attribute '_shared_roi_transform'

while cfg.MODEL.WEIGHTS = 'R-50.pkl' together with
cfg.merge_from_file('../configs/COCO-Detection/faster_rcnn_R_50_C4_1x.yaml') or cfg.merge_from_file('../configs/COCO-Detection/faster_rcnn_R_50_C4_3x.yaml') or cfg.merge_from_file('../configs/COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml')
result in

/github/py-bottom-up-attention/detectron2/layers/roi_align.py in forward(ctx, input, roi, output_size, spatial_scale, sampling_ratio, aligned)
     18         ctx.aligned = aligned
     19         output = _C.roi_align_forward(
---> 20             input, roi, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
     21         )
     22         return output

RuntimeError: Not compiled with GPU support (ROIAlign_forward at /github/py-bottom-up-attention/detectron2/layers/csrc/ROIAlign/ROIAlign.h:73)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fb5494f2193 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: detectron2::ROIAlign_forward(at::Tensor const&, at::Tensor const&, float, int, int, int, bool) + 0x171 (0x7fb54867df61 in /github/py-bottom-up-attention/detectron2/_C.cpython-36m-x86_64-linux-gnu.so)
frame #2: <unknown function> + 0x1faea (0x7fb54868daea in /github/py-bottom-up-attention/detectron2/_C.cpython-36m-x86_64-linux-gnu.so)
frame #3: <unknown function> + 0x1aba0 (0x7fb548688ba0 in /github/py-bottom-up-attention/detectron2/_C.cpython-36m-x86_64-linux-gnu.so)
frame #4: _PyCFunction_FastCallDict + 0x154 (0x5608c725ac54 in /opt/conda/bin/python)
frame #5: <unknown function> + 0x199abc (0x5608c72e2abc in /opt/conda/bin/python)
frame #6: _PyEval_EvalFrameDefault + 0x30a (0x5608c730575a in /opt/conda/bin/python)
frame #7: PyEval_EvalCodeEx + 0x329 (0x5608c72dd9b9 in /opt/conda/bin/python)
frame #8: <unknown function> + 0x1957d4 (0x5608c72de7d4 in /opt/conda/bin/python)
frame #9: PyObject_Call + 0x3e (0x5608c725aa5e in /opt/conda/bin/python)
frame #10: THPFunction_apply(_object*, _object*) + 0xa8f (0x7fb59454c82f in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #11: _PyCFunction_FastCallDict + 0x91 (0x5608c725ab91 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x199abc (0x5608c72e2abc in /opt/conda/bin/python)
frame #13: _PyEval_EvalFrameDefault + 0x30a (0x5608c730575a in /opt/conda/bin/python)
frame #14: _PyFunction_FastCallDict + 0x11b (0x5608c72dd2db in /opt/conda/bin/python)
frame #15: _PyObject_FastCallDict + 0x26f (0x5608c725b01f in /opt/conda/bin/python)
frame #16: _PyObject_Call_Prepend + 0x63 (0x5608c725faa3 in /opt/conda/bin/python)
frame #17: PyObject_Call + 0x3e (0x5608c725aa5e in /opt/conda/bin/python)
frame #18: _PyEval_EvalFrameDefault + 0x19e7 (0x5608c7306e37 in /opt/conda/bin/python)
frame #19: <unknown function> + 0x192e66 (0x5608c72dbe66 in /opt/conda/bin/python)
frame #20: _PyFunction_FastCallDict + 0x1be (0x5608c72dd37e in /opt/conda/bin/python)
frame #21: _PyObject_FastCallDict + 0x26f (0x5608c725b01f in /opt/conda/bin/python)
frame #22: _PyObject_Call_Prepend + 0x63 (0x5608c725faa3 in /opt/conda/bin/python)
frame #23: PyObject_Call + 0x3e (0x5608c725aa5e in /opt/conda/bin/python)
frame #24: <unknown function> + 0x16b371 (0x5608c72b4371 in /opt/conda/bin/python)
frame #25: _PyObject_FastCallDict + 0x8b (0x5608c725ae3b in /opt/conda/bin/python)
frame #26: <unknown function> + 0x199c0e (0x5608c72e2c0e in /opt/conda/bin/python)
frame #27: _PyEval_EvalFrameDefault + 0x30a (0x5608c730575a in /opt/conda/bin/python)
frame #28: _PyFunction_FastCallDict + 0x11b (0x5608c72dd2db in /opt/conda/bin/python)
frame #29: _PyObject_FastCallDict + 0x26f (0x5608c725b01f in /opt/conda/bin/python)
frame #30: _PyObject_Call_Prepend + 0x63 (0x5608c725faa3 in /opt/conda/bin/python)
frame #31: PyObject_Call + 0x3e (0x5608c725aa5e in /opt/conda/bin/python)
frame #32: _PyEval_EvalFrameDefault + 0x19e7 (0x5608c7306e37 in /opt/conda/bin/python)
frame #33: <unknown function> + 0x192e66 (0x5608c72dbe66 in /opt/conda/bin/python)
frame #34: _PyFunction_FastCallDict + 0x1be (0x5608c72dd37e in /opt/conda/bin/python)
frame #35: _PyObject_FastCallDict + 0x26f (0x5608c725b01f in /opt/conda/bin/python)
frame #36: _PyObject_Call_Prepend + 0x63 (0x5608c725faa3 in /opt/conda/bin/python)
frame #37: PyObject_Call + 0x3e (0x5608c725aa5e in /opt/conda/bin/python)
frame #38: <unknown function> + 0x16b371 (0x5608c72b4371 in /opt/conda/bin/python)
frame #39: _PyObject_FastCallDict + 0x8b (0x5608c725ae3b in /opt/conda/bin/python)
frame #40: <unknown function> + 0x199c0e (0x5608c72e2c0e in /opt/conda/bin/python)
frame #41: _PyEval_EvalFrameDefault + 0x30a (0x5608c730575a in /opt/conda/bin/python)
frame #42: <unknown function> + 0x193c5b (0x5608c72dcc5b in /opt/conda/bin/python)
frame #43: <unknown function> + 0x199b95 (0x5608c72e2b95 in /opt/conda/bin/python)
frame #44: _PyEval_EvalFrameDefault + 0x30a (0x5608c730575a in /opt/conda/bin/python)
frame #45: <unknown function> + 0x19329e (0x5608c72dc29e in /opt/conda/bin/python)
frame #46: <unknown function> + 0x193ed6 (0x5608c72dced6 in /opt/conda/bin/python)
frame #47: <unknown function> + 0x199b95 (0x5608c72e2b95 in /opt/conda/bin/python)
frame #48: _PyEval_EvalFrameDefault + 0x30a (0x5608c730575a in /opt/conda/bin/python)
frame #49: <unknown function> + 0x193c5b (0x5608c72dcc5b in /opt/conda/bin/python)
frame #50: <unknown function> + 0x199b95 (0x5608c72e2b95 in /opt/conda/bin/python)
frame #51: _PyEval_EvalFrameDefault + 0x30a (0x5608c730575a in /opt/conda/bin/python)
frame #52: PyEval_EvalCodeEx + 0x329 (0x5608c72dd9b9 in /opt/conda/bin/python)
frame #53: PyEval_EvalCode + 0x1c (0x5608c72de75c in /opt/conda/bin/python)
frame #54: <unknown function> + 0x1ba167 (0x5608c7303167 in /opt/conda/bin/python)
frame #55: _PyCFunction_FastCallDict + 0x91 (0x5608c725ab91 in /opt/conda/bin/python)
frame #56: <unknown function> + 0x199abc (0x5608c72e2abc in /opt/conda/bin/python)
frame #57: _PyEval_EvalFrameDefault + 0x30a (0x5608c730575a in /opt/conda/bin/python)
frame #58: _PyGen_Send + 0x256 (0x5608c72e5be6 in /opt/conda/bin/python)
frame #59: _PyEval_EvalFrameDefault + 0x144f (0x5608c730689f in /opt/conda/bin/python)
frame #60: _PyGen_Send + 0x256 (0x5608c72e5be6 in /opt/conda/bin/python)
frame #61: _PyEval_EvalFrameDefault + 0x144f (0x5608c730689f in /opt/conda/bin/python)
frame #62: _PyGen_Send + 0x256 (0x5608c72e5be6 in /opt/conda/bin/python)
frame #63: _PyCFunction_FastCallDict + 0x115 (0x5608c725ac15 in /opt/conda/bin/python)

@yezhengli-Mr9
Copy link

For RuntimeError: Not compiled with GPU support , I verified via
python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)'
in https://detectron2.readthedocs.io/tutorials/install.html#common-installation-issues suggested by detectron2 issue#1406 and it outputs True /usr/local/cuda.

This was referenced Jan 14, 2021
Open
Open
@zhangliyun9120
Copy link

@faizanahemad I wrote this up to extract the features for a detectron2 model. I believe this is correct and I have lots of comments. Let me know if this looks right to you:

class feature_extractor:
    '''
    Feature Extractor for detectron2
    '''
    def __init__(self, path = None, output_folder='./output', model = None, pred_thresh = 0.5):
        self.pred_thresh = pred_thresh
        self.output_folder = output_folder
        assert path is not None, 'Path should not be none'
        self.path = path
        if model == None:
            self.model = self._build_detection_model()
        else:
            assert model == detectron2.engine.defaults.DefaultPredictor, "model should be 'detectron2.engine.defaults.DefaultPredictor'"#
            self.model = model
            self.model.eval()
    def _build_detection_model(self):
        cfg = get_cfg()
        cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))
        cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
        cfg.SOLVER.IMS_PER_BATCH = 1
        cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1  # only has one class (pnumonia)
        #Just run these lines if you have the trained model im memory
        cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = self.pred_thresh   # set the testing threshold for this model
        #build model and return
        return DefaultPredictor(cfg)
    def _process_feature_extraction(self, img):#step 3
        '''
        #predictor.model.roi_heads.box_predictor.test_topk_per_image = 1000
        #predictor.model.roi_heads.box_predictor.test_nms_thresh = 0.99
        #predictor.model.roi_heads.box_predictor.test_score_thresh = 0.0
        #pred_boxes = [x.pred_boxes for x in instances]#can use prediction boxes
        '''
        torch.cuda.empty_cache()
        predictor = self.model
        with torch.no_grad():#https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html : _forward_box()
            features = predictor.model.backbone(img.tensor)#have to unsqueeze
            proposals, _ = predictor.model.proposal_generator(img, features, None)
            results, _ = predictor.model.roi_heads(img, features, proposals, None)
            #instances = predictor.model.roi_heads._forward_box(features, proposals)
        #get proposed boxes + rois + features + predictions
        proposal_boxes = [x.proposal_boxes for x in proposals]
        proposal_rois = predictor.model.roi_heads.box_pooler([features[f] for f in predictor.model.roi_heads.in_features], proposal_boxes)
        box_features = predictor.model.roi_heads.box_head(proposal_rois)
        predictions = predictor.model.roi_heads.box_predictor(box_features)#found here: https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html
        #WE CAN USE THE PREDICTION CLS TO FIND TOP SCOREING PROPOSAL BOXES!
        #pred_instances, losses = predictor.model.roi_heads.box_predictor.inference(predictions, proposals)#something to do with: NMS threshold for prediction results. found: https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/roi_heads/fast_rcnn.py#L460
        pred_df = pd.DataFrame(predictions[0].softmax(-1).tolist())
        pred_classes = pred_df.iloc[:,:-1].apply(np.argmax, axis=1)#get predicted classes
        keep = pred_df[pred_df.iloc[:,:-1].apply(lambda x: (x > self.pred_thresh)).values].index.tolist()#list of instances we should keep
        #start subsetting
        box_features = box_features[keep]
        proposal_boxes = proposals[0].proposal_boxes[keep]
        pred_classes = pred_classes[keep]
        probs = pred_df.iloc[keep, :].apply(lambda x: x[np.argmax(x)], axis=1).tolist()

        #['bbox', 'num_boxes', 'objects', 'image_width', 'image_height', 'cls_prob', 'image_id', 'features']
        #img.image_sizes[0]#h, w
        result = {
            'bbox': proposal_boxes.tensor.to('cpu').numpy(),
            'num_boxes' : len(proposal_boxes),
            'objects' : pred_classes.to_numpy,
            #'image_height': img.image_sizes[0][0],
            #'image_width': img.image_sizes[0][1],
            'cls_prob': np.asarray(probs),#needs to turn into vector!!!!!!!!!!
            'features': box_features.to('cpu').detach().numpy()
        }
        return result
            
    def _save_feature(self, file_name, feature, info):
        file_base_name = os.path.basename(file_name)
        file_base_name = file_base_name.split(".")[0]
        feature["image_id"] = file_base_name
        feature['image_height'] = info['height']
        feature['image_width'] = info['width']
        file_base_name = file_base_name + ".npy"
        np.save(os.path.join(self.output_folder, file_base_name), feature)
        
    def extract_features(self):#step 1
        torch.cuda.empty_cache()
        image_dir = self.path
        #print(image_dir)
        if type(image_dir) == pd.core.frame.DataFrame:#or pandas.core.frame.DataFrame. Iterate over a dataframe
            samples = []
            for idx, row in image_dir.iterrows():#get better name
                file = row['path']
                try:
                    features, infos = self.get_detectron2_features([file])
                    self._save_feature(file, features, infos[0])
                    samples.append(row)
                except BaseException:#if no features were found!
                    print('No features were found!')
                    pass
            df = pd.DataFrame(samples)
            #save final csv containing image base names, reports and report locations
            df.to_csv(os.path.join(self.output_folder, 'img_infos.csv'))
        elif os.path.isfile(image_dir):#if its a single file
            features, infos = self.get_detectron2_features([image_dir])
            self._save_feature(image_dir, features[0], infos[0])
            return features, infos
        else:#if its a directory
            files = glob.glob(os.path.join(image_dir, "*"))
            for idx, file in enumerate(files):
                try:
                    features, infos = self.get_detectron2_features([file])
                    self._save_feature(file, features, infos[0])
                except BaseException:
                    print('BaseException')
                    pass

    def get_detectron2_features(self, image_paths):#step 2
        #we have to PREPROCESS the tensor before partially executing it!
        #taken from https://github.com/facebookresearch/detectron2/blob/master/detectron2/engine/defaults.py
        predictor = self.model
        images = []
        image_info = []
        for image_path in image_paths:
            img = cv2.imread(image_path)
            height, width = img.shape[:2]
            img = predictor.transform_gen.get_transform(img).apply_image(img)
            img = torch.as_tensor(img.astype("float32").transpose(2, 0, 1))
            images.append({"image": img, "height": height, "width": width})
            image_info.append({"image_id": os.path.basename(image_path), "height": height, "width": width})
        imageList = predictor.model.preprocess_image(images)
        #returns features and infos
        return self._process_feature_extraction(imageList), image_info

Your input should be a DataFrame with the following columns = ['dicom_id', 'path', 'report_path', 'report', 'study_id', 'y']
Should look something like this:

fe = feature_extractor(path=outputs_df, output_folder='/path/to/your/numpy_files.npy')
fe.extract_features()

@kevkid I also get this problem:
AttributeError: 'StandardROIHeads' object has no attribute '_shared_roi_transform'

does above code work ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@kevkid @airsplay @faizanahemad @yezhengli-Mr9 @zhangliyun9120